Skip to content

[NGT] DNN for HP Pixel track selection - #51042

Merged
cmsbuild merged 5 commits into
cms-sw:masterfrom
EmanueleCoradin:pixel-track-HP-DNN
Aug 4, 2026
Merged

[NGT] DNN for HP Pixel track selection#51042
cmsbuild merged 5 commits into
cms-sw:masterfrom
EmanueleCoradin:pixel-track-HP-DNN

Conversation

@EmanueleCoradin

@EmanueleCoradin EmanueleCoradin commented May 26, 2026

Copy link
Copy Markdown
Contributor

PR description:

This PR introduces a DNN-based High-Purity (HP) pixel track selection for Phase-2 HLT pixel tracking using the PyTorch Alpaka package.

This PR is coupled with #51085, which introduces some changes and updates to the Patatrack pixel tracking.

Main changes:

  • Added PixelTrackTorchHighPuritySelector, an Alpaka EDProducer performing the selection.
  • Updated HLT Phase-2 pixel tracking sequences to use the DNN HP selector output.
  • Removed the Cut-Based selector from the Phase-2 pixel tracking sequence.

Expected impact:

  • Enables vertex-independent HP pixel-track selection using DNN inference.
  • Simplifies the Phase-2 HLT heterogeneous pixel tracking workflow by removing the circular dependency between vertexing and HP track selection.

Related material:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Not a backport.

@cmsbuild

cmsbuild commented May 26, 2026

Copy link
Copy Markdown
Contributor

cms-bot internal usage

@EmanueleCoradin

Copy link
Copy Markdown
Contributor Author

type ngt

@cmsbuild cmsbuild added the ngt label May 26, 2026
@cmsbuild

Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51042/49484

@cmsbuild

Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51042/49485

@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #51042 was updated.

@mmusich

mmusich commented May 27, 2026

Copy link
Copy Markdown
Contributor

test parameters:

  • enable = hlt_p2_integration, hlt_p2_timing
  • workflows = ph2_hlt

@mmusich

mmusich commented May 27, 2026

Copy link
Copy Markdown
Contributor

@cmsbuild, please test

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Failed Tests: RelVals
Size: This PR adds an extra 84KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2a8974/53493/summary.html
COMMIT: f138199
CMSSW: CMSSW_17_0_X_2026-05-26-2300/el8_amd64_gcc13
Additional Tests: HLT_P2_INTEGRATION,HLT_P2_TIMING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/51042/53493/install.sh to create a dev area with all the needed externals and cmssw changes.

HLT P2 Timing: chart

Failed RelVals

----- Begin Fatal Exception 27-May-2026 09:36:23 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'nanoAOD_step'
   [2] Calling method for module SimplePixelTrackSoATabFlatTableProducer/'hltPixelTrackSoATable'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for a container with elements of type: PixelTrackSoATab
Looking for module label: hltPhase2PixelTrackSoATableProducer
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------
  • 34434.75334434.753_TTbar_14TeV+Run4D121_HLT75e33TimingLegacyTracking/step2_TTbar_14TeV+Run4D121_HLT75e33TimingLegacyTracking.log
  • 34434.773
----- Begin Fatal Exception 27-May-2026 09:41:41 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'nanoAOD_step'
   [2] Calling method for module SimplePixelTrackSoATabFlatTableProducer/'hltPixelTrackSoATable'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for a container with elements of type: PixelTrackSoATab
Looking for module label: hltPhase2PixelTrackSoATableProducer
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------
Expand to see more relval errors ...

@mmusich

mmusich commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

code-checks

@cmsbuild

Copy link
Copy Markdown
Contributor

@EmanueleCoradin

Copy link
Copy Markdown
Contributor Author

code-checks

@cmsbuild

Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51042/50368

@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #51042 was updated. @Martin-Grunewald, @Moanwar, @cmsbuild, @ctarricone, @davidlange6, @fabiocos, @ftenchini, @gabrielmscampos, @jfernan2, @mandrenguyen, @mmusich, @rseidita, @srimanob can you please check and sign again.

@EmanueleCoradin

Copy link
Copy Markdown
Contributor Author

I am running some tests locally to check the PR before launching the bot again

@EmanueleCoradin

Copy link
Copy Markdown
Contributor Author

@cmsbuild please test

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Failed Tests: RelVals-AMD_MI300X
Size: This PR adds an extra 60KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2a8974/54996/summary.html
COMMIT: bd2f0d8
CMSSW: CMSSW_20_1_X_2026-07-28-2300/el9_amd64_gcc13
Additional Tests: GPU,HLT_P2_INTEGRATION,HLT_P2_TIMING,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51042/54996/install.sh to create a dev area with all the needed externals and cmssw changes.

HLT P2 Timing: chart

Failed RelVals-AMD_MI300X

  • 34434.40334434.403_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Validation/step2_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Validation.log

Comparison Summary

Summary:

  • You potentially removed 2 lines from the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 11 differences found in the comparisons
  • DQMHistoTests: Total files compared: 61
  • DQMHistoTests: Total histograms compared: 4324517
  • DQMHistoTests: Total failures: 41406
  • DQMHistoTests: Total nulls: 76
  • DQMHistoTests: Total successes: 4283017
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: -17.188 KiB( 60 files compared)
  • DQMHistoSizes: changed ( 34434.7503 ): -17.188 KiB HLT/HeterogeneousMonitoring
  • Checked 257 log files, 213 edm output root files, 61 DQM output files
  • TriggerResults: found differences in 18 / 60 workflows

NVIDIA_H100 Comparison Summary

Summary:

  • No significant changes to the logs found
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 83 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 172823
  • DQMHistoTests: Total failures: 12936
  • DQMHistoTests: Total nulls: 4
  • DQMHistoTests: Total successes: 159883
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: -17.188 KiB( 6 files compared)
  • DQMHistoSizes: changed ( 34434.7503 ): -17.188 KiB HLT/HeterogeneousMonitoring
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: found differences in 6 / 6 workflows

NVIDIA_L40S Comparison Summary

Summary:

  • No significant changes to the logs found
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 65 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 172823
  • DQMHistoTests: Total failures: 12316
  • DQMHistoTests: Total nulls: 4
  • DQMHistoTests: Total successes: 160503
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: -17.188 KiB( 6 files compared)
  • DQMHistoSizes: changed ( 34434.7503 ): -17.188 KiB HLT/HeterogeneousMonitoring
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: found differences in 6 / 6 workflows

NVIDIA_T4 Comparison Summary

Summary:

  • No significant changes to the logs found
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 86 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 172823
  • DQMHistoTests: Total failures: 13606
  • DQMHistoTests: Total nulls: 4
  • DQMHistoTests: Total successes: 159213
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: -17.188 KiB( 6 files compared)
  • DQMHistoSizes: changed ( 34434.7503 ): -17.188 KiB HLT/HeterogeneousMonitoring
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: found differences in 6 / 6 workflows

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 5 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34434.77_TTbar_14TeV+Run4D121_NGTScouting step2 max memory diff 55.2 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.771_TTbar_14TeV+Run4D121_NGTScoutingAll step2 max memory diff 55.2 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.772_TTbar_14TeV+Run4D121_NGTScoutingWithNano step2 max memory diff 55.8 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.773_TTbar_14TeV+Run4D121_NGTScoutingWithNanoValid step2 max memory diff 55.8 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.774_TTbar_14TeV+Run4D121_L1NGTScoutingWithNanoValid step2 max memory diff 55.2 exceeds +/- 30.0 MiB

Max Memory Comparisons exceeding threshold NVIDIA_H100

@cms-sw/core-l2 , I found 7 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34434.402_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka step2 max memory diff 328.5 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.403_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Validation step2 max memory diff 328.6 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling step2 max memory diff 327.9 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.712_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnly step2 max memory diff 329.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.713_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnlyAlpakaValidationLST step2 max memory diff 328.8 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7503_TTbar_14TeV+Run4D121_HLTHeterogeneousValid step2 max memory diff 329.9 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.751_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka step2 max memory diff 328.1 exceeds +/- 30.0 MiB

Max Memory Comparisons exceeding threshold NVIDIA_L40S

@cms-sw/core-l2 , I found 7 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34434.402_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka step2 max memory diff 457.6 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.403_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Validation step2 max memory diff 456.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling step2 max memory diff 456.6 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.712_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnly step2 max memory diff 457.3 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.713_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnlyAlpakaValidationLST step2 max memory diff 457.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7503_TTbar_14TeV+Run4D121_HLTHeterogeneousValid step2 max memory diff 456.3 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.751_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka step2 max memory diff 456.4 exceeds +/- 30.0 MiB

Max Memory Comparisons exceeding threshold NVIDIA_T4

@cms-sw/core-l2 , I found 7 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34434.402_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka step2 max memory diff 338.9 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.403_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Validation step2 max memory diff 338.9 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling step2 max memory diff 338.8 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.712_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnly step2 max memory diff 339.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.713_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnlyAlpakaValidationLST step2 max memory diff 339.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7503_TTbar_14TeV+Run4D121_HLTHeterogeneousValid step2 max memory diff 588.9 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.751_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka step2 max memory diff 338.9 exceeds +/- 30.0 MiB

@mmusich

mmusich commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

-1

gpu relvals on amd are unfortunately broken since a while for independent reasons.

@mmusich

mmusich commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

test parameters:

  • enable_tests = hlt_p2_integration, hlt_p2_timing
  • workflows = ph2_hlt

@mmusich

mmusich commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

@cmsbuild, please test

  • let's try without

@cmsbuild

Copy link
Copy Markdown
Contributor

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2a8974/55003/summary.html
COMMIT: bd2f0d8
CMSSW: CMSSW_20_1_X_2026-07-29-1100/el9_amd64_gcc13
Additional Tests: HLT_P2_INTEGRATION,HLT_P2_TIMING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51042/55003/install.sh to create a dev area with all the needed externals and cmssw changes.

HLT P2 Timing: chart

Comparison Summary

Summary:

  • You potentially removed 1 lines from the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 11 differences found in the comparisons
  • DQMHistoTests: Total files compared: 61
  • DQMHistoTests: Total histograms compared: 4324517
  • DQMHistoTests: Total failures: 41398
  • DQMHistoTests: Total nulls: 76
  • DQMHistoTests: Total successes: 4283025
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: -17.188 KiB( 60 files compared)
  • DQMHistoSizes: changed ( 34434.7503 ): -17.188 KiB HLT/HeterogeneousMonitoring
  • Checked 257 log files, 213 edm output root files, 61 DQM output files
  • TriggerResults: found differences in 18 / 60 workflows

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 5 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34434.77_TTbar_14TeV+Run4D121_NGTScouting step2 max memory diff 55.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.771_TTbar_14TeV+Run4D121_NGTScoutingAll step2 max memory diff 55.2 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.772_TTbar_14TeV+Run4D121_NGTScoutingWithNano step2 max memory diff 55.8 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.773_TTbar_14TeV+Run4D121_NGTScoutingWithNanoValid step2 max memory diff 55.8 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.774_TTbar_14TeV+Run4D121_L1NGTScoutingWithNanoValid step2 max memory diff 55.2 exceeds +/- 30.0 MiB

@mmusich

mmusich commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

+hlt

@mmusich

mmusich commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

@cms-sw/reconstruction-l2 @cms-sw/dqm-l2 please review

@gabrielmscampos

Copy link
Copy Markdown
Member

+dqm

Comment thread RecoTracker/FinalTrackSelectors/interface/PixelTrackFeaturesSoA.h
Comment thread RecoTracker/FinalTrackSelectors/interface/PixelTrackFeaturesSoA.h
@Moanwar

Moanwar commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

+1

@cmsbuild

cmsbuild commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @ftenchini, @sextonkennedy, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)

@mandrenguyen

Copy link
Copy Markdown
Contributor

+1

@mmusich

mmusich commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

For the record, I think this PR broke the unit test test_heterogeneousMenu (introduced at #51583), see logs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants